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The Quality of Experimental Methodology in 
Counseling and Counselor Education 

The aim of science above all else is to discover new and useful 
information in the form of verifiable data, that is, of data 
obtained under conditions such that other qualified people can 
make similar observations and obtain the same results. This 
calls for orderliness and precision in uncovering relationships 
and communicating them to others. (Hilgard, 1962, p. 9) 

Counseling psychology is usually considered an applied science, and pre- 
sumably the aim of science as stated above is also a goal for this branch of 
psychology. Seme counseling psychologists (Hansen & Warner, 197^; Thoresen, 
1969; ^^iteley, 1967) have questioned whether the profession is making signi- 
ficant progress toward this goal. An often' read criticism is that the overall 
quality of research is questionable. Members of the profession (Pawlicki, 1970; 
Schmidt & Pepinsky, 1965; Thoresen, 1969) have specified the need for a con- 
serted effort toward the improvement of the quality of the profession's research. 

There are obvious reasons which require that the concern for quality 
research be carefully considered. One^ is that the profession may risk building 
a research base on a foundation of sand. With the retic^ace of most profession- 
als to rep7^cate studies, the findings of published research are not often 
substantiated by empirical methods. Consequently, research based on prior 
findings and on the assumption that published research is established fact may 
perpetuate questionable conclusions. A second rp;qson for the concern for 
quality research is the waste of professional time and effort invested in 
research with poor methodology. The progress of counseling to establish a re- 
search base will be inhibited by design and methodological errors. "Research 
which is not well formulated is more than worthless since it becomes deceptive 
as well (Whiteley, 1967, p. 281)." 

While there is consensus of the literature that a lack of well-planned and 

executed research in counseling exists, the authors of such conclusions base 

\ 

their comments on varying types of data. Some communicate intuitive estimations 



of the state of counseling research (Coleman, 1957; Fisher & Roth, 1961), while 
others conclude from estimations following a systematic review of the literature 
on various topics (Gazda & Larsen, 1968; Hansen & Warner, 1971; Pawlicki, 1970; 
Schmidt & Pepinsky, 1965). Though several authors have surveyed counseling 
journals to describe such variables as statistics used in data analysis and 
institutional affiliation of authors, there has been only one published study 
to date which has attempted to estimate the quality of research methodology by 
empirical means. Kelley, Smits, Leventhal, and Rhodes (1970) systematically 
evaluated the designs of all empirical studies published in the Journal of ^ 
Counseling Psychology from 1964 through 1968. Using Campbell and Stanley's ^ 
(1963) criteria, they concluded that this group of studies "has little relevance 
beyond that of generating testable hypotheses (p. 340)." The majority of studies 
had sources of invalidity not controlled^ in their designs. This evaluation, 
however, scrutinized only one aspect of research methodology, that of design. 

The purpose of the present study was to empirically estimate the quality 
of published experimental research in the fields of counseling and counselor 
education and to determine whether there has been an improvement over time of 
that (fuality. Specifically, the intent of the investigator was to systematically 
evaluate the methodology of experimental research which has been published from 
1962 through 1973 in counseling and counselor education journals. This 
investigator recognizes that the study has examined only one of the two issues 
of quality of research. The research irethodology has been evaluated, while 
relevance of the results and studied phenomena to the profession has not been 
examined, labile neither is a sufficient condition, both are necessary conditions 
for quality research in a profession. 

Method 

Sample 

The sample of 152 studies was randomly chosen from the experimental studies 

'i 
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published from 1962 through 1973 in the three major counseling and counselor 
education journals: Journal of Counseling Psycholog y, Personnel and Guidance 
Journal, and Counselor Education and Sipervision, The three journals were 
chosen as the major publication outlets for experimental studies for counselor 
researchers and educators. Thirty-eight studies were randomly chosen from 
each of these year spans: 1962-1964, 1965-1967, 1968-1970, 1971-1973. The 
total sample size represented 41 percen^t of the population. 

Design 

The independent variable of interest was years of published research 
in counseling and counselor education. The total time span of 1962 through 
1973 was considered. This was divided into four levels, feach level containing 
three years: 1962-1964; 1965-1967; 1968-1970; 1971-1973. Thus, the design of 
this correlational study is a 1 x 4 matrix wi^h .an equal number of observations 
per cell: 



Years 





^2 


^3 


^4 


ni=38 


n2=38 


n3=38 


n4=38 



71 = 1962 - 1964 

72 = 1965 - 1967 
y3 = 1968 •- 1970 
y^^ = 1971 - 1973 



ERJC 



The hypotheses tested were: 

1. There is a significant linear trend for the dependent measures across 
the four year spans. 

2. There is a significant quadratic trend for the dependent measures 
across the four year spans. 

3. There is a significant 'residual trend for the dependent measures 
across the four year spans. 

Instrument 

Assessment of the reporting and methodology of the studies was done using 
the Evaluation Instrument for Experimental Methodology (EIEM) , a rating form 



developed by the investigator. The instrument was constructed by a compilation^, 
of the recurring problems in experimental coui\seling resear^^h cited in the 

A 

professional literature. Special attention was also given to published guides 
for the evaluation of research. Experts in research methodology were consulted 
during the initial and trial stages of instrument development. 

The EIEM has 37 Likert-scaled items, each item having six response options. 
The six responses were consistent across items (see Figure 3 ). Six scores were 
derived from the EIEM: reporting, introduction, methods, results, discussion 
and total. Concurrent validity of the instrument, was estimated to be .85, and 
the interrater reliability estimate prior to data collection was .79 using 
Hoyt*s Analysis of Variance. During the data collection the average interrater 
reliability coefficient was .78. 
Procedures 

Fifteen of tW^ total of 152 studies were ^.randomly chosen to be independently 
r^t-ed by all of the raters in order to establish interrater reliability estimates. 
The remaining 137 were randomly ordered and then assigned to the three raters. 
The fifteen studies designated for reliability checks were evenly placed through- 
out the sequence of the other studies for each rater. The random sequence of 
the studies was intended to avoid a time or fatigue bias, and the random 
assignment to rater was done to avoid a rater bias. Prior to the rating process 
each study was photocopied and blinded for journal name, author^s name and 
affiliation, and dates. 

Three doctoral students, recognized as having expertise in research method- 
ology and statistics, were paid to rate the studies. They were trained in the 
use of the EIEM immediately prior to the rating process. During the two week 
process each rater work independently. Interrater reliability checks were made 
at fifteen points throughout the process to establish that a reliability of at 
least .70 was maintained. This goal was accomplished. 

O 
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Resul ts 

•Mean scores and standard deviations for the four groups , on 
the five subscales and one total score on the EIEM are shown in 
Table 1, The standard deviations reported are those used in the 
data analysis following a correction for having a finite popula- 
tion • 

Table 1. Mean Stores and Corrected Standard Deviations for the 
ScalesJ of the EIEM 
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A multivariate analysis of orthogonal polynomials was accom- 
plished to determine^ the form of the relationship between the year 
spans for the six dependent variables. The purpose of this ana- 
lysis, commonly called a trend analysis, was to determine whether 
the means of the dependent variables were influenced by change in 
the independent variable, time or years. The results of the ana- 
lysis are found in Table 2. 
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Table 2. Multivariate Test for Orthogonal Polynomials 
for Six Scales of EIEM 



Test F-ratio df 



Linear 2 . 135 1 ,148 ' .05 

Quadratic ' . 320 1,148 - <.93- 

Residual. i:043 1,148 <.40 

A significant linear relationship with a nonzero slope was ^ 
found to exist across time. After graphing the observed and es-- 
timated means for each dependent, measure (Figures 1 and 2) , a 
slightly positive significant linear trend was evddent. There- 
fore, the quality of methodology in counseling and counselor 
education has improved over the twelve- years^-^ However, the degree 
of increase is slight. The estimated slopes, each defined as the 
increase in the mean of the dependent variable from one year ^'^pan 
to the next, vary from .08 to .17 on the 1-6 scale used' for the 
EIEM. ^ 

Although prediction into time should be made with caution, 
the trend based on experimental research published from 1962 
through 1973, if maintained at the same rate, will predict a mean 
rating for the total score for quality of 'methodology and report- 
ing of 4.46 by 1974 , 4. 94 bv 1991, and 5.30 by '2000. Bv 1994 the 
mean will indicate that in an overall evaluation experimental 

\ 
I 

research in counseling and counselor edulcatlon has become cleariy 

( 

adequate. ^ 
Univariate and multivariate confidence intervals were gener- 
ated around the estimated means of the dependent variables to 
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considar the present state of experimental research In the pro- 
fession. For this evaluation only the most recent year span, 1971 
to 1973 , was considered, Fij>ure 3 contains the graphite represent- 
ation of the univxir^iate intervals compared to the \s.cale of the 
dependent measures deriveM from the CIEM. Two subscales, report- 
ing and introduction, are clearly on the adequate end of the 
scale, while the three other subscales,, method, results, and dis- 
cussion, span the middle area of the scale. The quality of the 
reporting and the" introduction section was " leariv adequ;iLc** for 
the last vear span. However, the measures which evaluated th<: es- 
sence of the experimental ilesearch were considerably lowor. 
These aspects of the evaluated research studies were in the ;'ra 7 
area, neither ''clearly inadequate'Vnor "clearly adequate," The 
interval for the total score was predictably between the two 
groupings and could be characterized^ as "barely adequate," 

Discussion I 
The results of the eval|iation of experimental stuxiies in 
cbuseling and counselor education support both positive and 
nep.ative speculations of the quality of the profession's research. 
The results ind^icate that the re were differences between the year 
spans in the form of an increasing linear trend. This increasing 
linear trend is the major finding of the investigation. It in- 
dicates that the quality of methodology used in published counseling 
research Is improving. However, the rate of improvement is minimal, 
and prediction over time is risky. Contributing factors to the 
quality of published research are complex and probably do not act 
uniformly over time. 
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Therefl^re, the postulation by several c jnselor educators 
(Carkhuff, 1965; Myers, 1966; Patterson, 1963) that the quality 

• * , 

of research In counseling and counselor education has been im- 
proving over time is supported by this empirical investigation 
of methodology, but with the cautions previously stated. The 
results' of this study, however, are applicable only to the 
population of experimental studies of counseling and counselor 
education re sear ch • / The conclusions of other professionals 
(Gazda & Larsen, 1968; Hansen & Warner, 1971-; Thorensen, 1969) 
that there-^s^ a lack of well-planned and executed research is 
also supported, as demonstrated by the examination of the con- 
fidence intervals .for* the dependent measures for the years 
1971-1973. The quality of reporting for recent experimental 
publications is relatively high, as evidenced by a mean of 4.89^ 
for the overall rating of reporting and by the confidence interval 
for the dependen*t variable "reporting. How^ever, the quality of 
methodology was rated less than '^barely Adequate." The mean for 
the overall methodology . rating W2S 3.87/ While the quality of 
the reporting of an experimental project is important for repli- 
cation and communication within the -profession, the impact of 
poor quality methodology is greater. Misleading or false results, 
a possible consequence of poor methodology, can be costly, especially 
in fields that deal with human beings. 

In an effort to provide feedback on specific aspects of 
experimental me tho doilogy , the means for individual items were 
examined. The low ratings of the items on random selection and 
discussion items are particularly | significant . It is probable 
that m-ny readers, especially those with inadequate knowledge of 



research methodology, look to the discussion section for 
conclusions without careful consideration of the previous 
sections of the experimental report. Thus, considering the 
rated inadequacy' of^ generalization statements , too many re- \ 
searchers are misrepresenting the applicability of thejr re-* 
suits, and too- many consumers may not be perceiving the il- 
legitimate generalizations. Such occurrences are potentially 
harmful to the profession, and thus, also to those who are y/ 
served. The continued growth of counseling research is also 
hampered by such practices. 

This study ^s reults can be used in several ways: as a 
baseline of the status of ^published counseling research at a 
given point in time: as an attentional device directed at the 
need for more carefully executed research; as an educational 
tool for those who read and evaluate professional research; as 
an educational tool for those who teach research skills for 
improvement and/or reemphasis of course content; and as feed- 
back to editors of journals for improvement in review and ac- 
ceptance criteria. 

Implications from the results of this study are relevant to 
all coi; iselors and counselor educators, x^hether they are re- 
searchers, practitioners, editors, or educators, as well as 
consumers of published research. Recommendations to researchers 
on the specific aspects of reports which contribute to questionabl 
and deceptive experimental results have been made elsewhere 
(Ripstra, 1974). An additional suggestion derived from the re- 
sults is that researchern as well as editors should consider the 



replication of previous research as valuable professional effort. 
Replication is a necessary contributor to the development of a 
reliable research base for a profession, but unfortunately it has 
not been a type of research that results in reinforcement for 
the researcher. Replication would be one method of increasing the 
pace of impro vftment. of the quality of counseling research. 

The impilication for educators is to consider that counselors 
55hould be trained to be competent readers and evaluators of re- 
search. A counselor training prograin r.hould include a research 
methodology unit i-zhich would teach the trainee to discriminate 
quality of riethot!ology • For those educators w'lo teach research 
skills the specific result*? of the items of the EIEM pinpoint 
the nrca<? of me th^doloj^y which contribute to the less-than-adequac 
evaluation of current counseling research (see ^.ipstra, 1974), Th 
KIEM or some similar instrument could be used as a learning tool 
for the trainee. 

The rccommcnc'atioii for the practitioner and consumer of 
counseling research is to carefully consider all aspects of a 
research report. The results of the evaluation of experimental 
studies showed ttiat the methods, results, and discussion sections 
of a report have the highest probabilities for error or misleading 
statements. These sections also have the largest impact on the 
signif icatxce of the results of an experimental study. Careful 
reading of published studies is therefore imperative. The reader' 
assumption that a study which has been published in a professional 
journal has quality "methodology and states legitimate results and 
conclusions is not entirely justified. 
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In conclusion, the systematic evaluation of tntithodology and 
reporting of research^in counseling aad counselor education revealed 
mixed results. The quality of reporting was quite adequate, while 
the quality of experimental methodology was barely mediocre. Des- 
pite the trend of increasing quality* the research in these fields 
must be viewed critically. Whiteley^s (1967) comment that poorly 
formulated research' is not only worthless but deceptive should be 
heeded by counseling researchers, editors, and research consumers 
in an effort to upgrade the profession's research, protect future 
clients and trainees, and promote better counseling, service and 



train iug . 
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